UDM: User Direct Messaging for General-Purpose Multiprocessing

User Direct Messaging (UDM) allows user-level, processor-to-processor messaging to coexist with general multiprogramming and virtual memory. Direct messaging, where processors launch and receive messages in tens of cycles directly via network interface FIFOs as opposed to indirectly via memory, offers high message bandwidth and low delivery latency by avoiding memory delay and buffer management overhead. However, user-level direct messaging implementations to date are limited in that they operate only in single-user machines or with strict gang scheduling. In this paper, we develop a messaging approach for protected, direct delivery with a single, unified user interface but with an underlying implementation that provides two delivery cases: a fast, common case corresponding to direct user access to hardware queues and a second case using virtual buffering that is invoked transparently when required by the demands of multiprogramming, virtual memory or user intransigence. The paper lays out a simple, efficient messaging model for user direct messaging that allows both user interrupts and user polling by explicitly incorporating atomicity. The paper then identifies two mechanisms that enable the model to map to a fast, hardware path: a revocable interrupt disable mechanism in hardware permits the user to block the network in a limited way and an overflow control scheme used in the virtual buffer case allows all buffer management overhead to be avoided in the fast path. Experiments with real and synthetic applications on an existing, single-user machine, Alewife, and a new, simulated, multi-user machine, FUGU, show that the cost of the fast case is within a few cycles of the cost of unprotected, kernel messaging and indicate that the fast case can indeed be expected to be the common case under ordinary conditions.